Computer software user interfaces are designed to be operated by human beings. Like so many tasks, controlling user interfaces can be simple for a live person, but difficult to automate. However, there are cases when it is desirable for an automated process to navigate a user interface without the help of a person. One case is an automated tool that interacts with third party user interfaces, for example on the Web. An example of such a system is a program that automatically searches the Web for malware (e.g., viruses, Trojan horses, etc.). Such a system would need to download and install many different software packages in order to examine their contents. Different Web based installation programs can have widely different user interfaces. Thus, it would be desirable if the system could automatically operate a wide variety of user interfaces.
Another case is an automated testing tool that needs to drive a user interface in order to test the underlying software. As developers create new software, it is common to do a lot of automated testing in this manner. Because the user interface can change between builds, it would be desirable for the automated test tool to be able to drive various user interface configurations and formats.
One way for an automated process to drive user interfaces is to build the automated process such that it includes specific information concerning the user interface it is to operate. However, this significantly limits the utility of the automated process. For example, it would be desirable for the above-described Web searching system to be able to operate the interfaces of the different installation programs it encounters on the Web, concerning which it would have no previous information. In a similar vein, if the automated testing tool must be build such that it includes detailed information about a user interface it is to operate, it would have to be re-scripted every time the user interfaces changes. These problems are made even more complex by the wide variety of technologies used to implement contemporary user interfaces, such as Win32, Java, variations of HTML, Flash, etc.
Application Programming Interfaces (APIs) exist that can be employed in a software user interface to allow automated processes to interrogate and drive it. An example of such an automation API is Microsoft Active Accessibility. However, user interfaces are not required to support such APIs, and as a result most user interfaces do not. There is no existing mechanism that allows an automated process to operate a user interface that does not support an automated API, unless the automated process has prior information of the user interface layout. It would be desirable to address these issues.